The text search query sql${documentsTable.processed_content_jieba_tokens} @> ${content.text}`` is incorrect. The content.text is a raw string, but a `jsonb` containment query (`@>`) on `processed_content_jieba_tokens` expects a JSONB array of tokens. The input text should be tokenized first (e.g., using Jieba), and then the tokens should be passed to the SQL query as a JSON array string, for example: `sql`... @> ${JSON.stringify(tokens)}::jsonb`.

The vector similarity operator <=> is being used in a WHERE clause. This operator calculates the distance and is typically used in an ORDER BY clause to sort results by similarity. Using it in a WHERE clause without a comparison is incorrect. You should use orderBy to sort by distance.

The retrieveDocuments function performs two separate database queries for text and vector search, and then concatenates the results. This approach has several issues:

• Duplicates: It can return duplicate documents if a document matches both text and vector queries.
• Incorrect Pagination: The limit and offset are applied to each query separately, not to the combined result set. This means you could get up to 2 * limit results, and pagination will not work as expected.
• Inconsistent Ranking: The results from the two queries are not ranked against each other; they are simply appended. This can lead to less relevant results appearing before more relevant ones.

A better approach would be to combine these into a single query if possible, or fetch results and then merge, de-duplicate, and re-rank them in the application before applying pagination.

The foreign key on chat_messages_id defaults to ON DELETE NO ACTION. This will cause an error when trying to delete a chat message that has associated documents, which can lead to data integrity issues or unexpected application behavior. If documents should be deleted along with their corresponding message, consider using onDelete: 'cascade'.

The regex https?:\/\/\S+ for extracting links is a bit too greedy. It can incorrectly include trailing punctuation like commas, periods, or closing parentheses as part of the URL, which can lead to failed scraping attempts. Consider cleaning the matched URLs to remove such trailing characters.

The URL for the Jina reader service is hardcoded. It's better to extract this into a constant or a configuration variable to make it easier to change or update in the future.

The scraped document is used for rawContent, processedContent, and summary without any processing. Also, processedContentJiebaTokens is an empty array. For these fields to be effective for search and retrieval, the content should be processed (e.g., cleaned, tokenized) and a summary should be generated. This implementation seems like a placeholder and should be enhanced for the feature to be fully functional.

The recordDocuments function in packages/core/src/models/documents.ts uses onConflictDoUpdate with chat_messages_id as the target. This implies that chat_messages_id should be unique. To enforce this at the database level and for clarity, you should add a unique index on this column.